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ABSTRACT 

The Massachusetts Workplace Literacy Consortium sought to 
upgrade work-related literacy skills at 22 partner sites in the state. 

Members included manufacturers, health care organizations, educational 
institutions, and labor unions. In its third year, the consortium served 
1,179 workers with classes in English for speakers of other languages, adult 
secondary education/General Educational Development, and adult basic 
education. Findings of an evaluation of the third year included the 
following: (1) consortium members felt the Planning and Evaluation Team (PET) 

model provided many benefits, such as cost effectiveness, resource sharing, 
and communication; (2) confusion over roles of the state department of 
education and program coordinators hindered the first 2 years; (3) PET was 
considered critical to program success because it kept stakeholders informed, 
enabled open discussion, and promoted stakeholder buy-in; (4) obstacles to 
student participation in PET were difficulties with English, program 
completion, and a rotation plan that contributed to high turnover; (5) 
underrepresentation of important stakeholders was a serious threat to PET; 
and (6) PET was considered helpful but not sufficient to ensure that the 
program was institutionalized at all sites. (Appendices contain the 
following: program standards and quality indicators; evaluation interview 
questions; list of PET site visits; PET participation chart; and PET survey 
and meeting observation forms.) (SK) 
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Executive Summary 



The Massachusetts Workplace Literacy Consortium (MWLC), Wave 6 was funded for 
three years from November 1, 1994 to October 31, 1997 through the National Workplace 
Literacy Program (NWLP). The MWLC was administered through the Adult and 
Community Learning Services Cluster of the Massachusetts Department of Education and 
seven educational providers. Members of the Consortium included manufacturing 
companies, health care organizations, educational institutions, and labor unions. In its 
third year, the MWLC provided workplace education services to upgrade work-related 
literacy skills at twenty-two partner sites throughout the Commonwealth of 
Massachusetts. 

During the third year of the project, the MWLC administered 69 classes serving 572 
student workers in Period 5 and 63 classes serving 602 students in Period 6. The primary 
type of content taught was ESOL, followed by ASE/GED and ABE 

The focus of the Year 3 evaluation was on a key aspect of the original MWLC concept: 
the planning and evaluation team model. In particular, the focus of the current evaluation 
was on the organization and operation of planning and evaluation teams at the consortium 
and partnership site levels. An additional piece of the Year 3 evaluation was a summary 
of work on a key component of the project: the Indicators of Quality for Workplace 
Literacy Programs. Information about students served in the MWLC is presented in the 
MWLC: Annual Performance Report and MWLC: Semi-annual Performance Report. 

Highlights of Evaluation Questions and Findings 

Consortium Planning and Evaluation Team 

i 

t 

II 

• MWLC members felt that they were part of a team working toward a single goal: the 
improvement of workplace literacy programs in Massachusetts: They also indicated 
that the MWLC model provided many benefits over working alone. 

• The major advantages of the MWLC model according to project coordinators were a) 
the cost effectiveness of the model, b) the ability to share resources and draw' on the 
resources of the state and other programs, and c) the opportunit es for communication 
with colleagues. 

• Coordinators did not identify any disadvantages of the MWLC model. 

• At the beginning of the project there was not a clear understanding of the roles of the 
Department of Education, program coordinators, or the CPET i :self in governing the 
program. This resulted in the CPET consuming a significant amount of human and 
time resources over the first two years of the program in defining and accepting their 
roles. 
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Planning and Evaluation Teams 



• There was no evidence of explicit or implicit barriers to the participation of any 
members of the PET with the possible exception of student workers. Respondents 
indicated that all other PET members were able to fully participate in discussions and 
decision-making. 

• Members felt that the PET was critical to the success of a workplace literacy program 
because the PET a) provides the opportunity to keep all stakeholders informed on a 
regular basis, b) presents a forum for open discussion, c) strengthens the partnership 
and team approach within the organization, and d) promotes buy-in from all 
stakeholders. 



• Attendance records indicated that most current PET members attend nine or ten 
meetings per year. 

• The greatest obstacles to student participation in the PET were difficulties in 
expressing themselves in English and a high rate of turnover on the PET due to a) 
students’ completing the program, or b) a planned system of rotating students through 
the PET. 

• ‘ Under-representation of important stakeholders such as company management, 

supervisory staff, or students was seen as the most serious threat to the PET. 

• A properly functioning PET was identified as necessary but not sufficient to ensure 
the success and institutionalization of a workplace literacy program. The decision to 
continue the workplace literacy program is often based on factors other than the 
success of the program. 

Quality > Indicators 

• Institutionalization is often not a quality issue. Initial decisions whether to 
institutionalize a program are often based on whether there is money available in the 
budget to continue the program. 

• There are many factors which affect the quality of a program ovei which the 
education provider has little, if any, control. It is necessary to define standaids foi all 
stakeholders that address the expectations of the entire program. 



? 
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Section I: Introduction 



Section I: Introduction 

Background 

The Massachusetts Workplace Literacy Consortium (MWLC) was funded for three years 
from November 1, 1994 to October 31, 1997 through the National Workplace Literacy 
Program (NWLP). The MWLC was administered through the Adult and Community 
Learning Services Cluster of the Massachusetts Department of Education (DOE) and 
seven education providers representing business, labor, and education. Its members 
included manufacturing companies, health care organizations, educational institutions, 
and labor unions. In its third year, the MWLC provided workplace education services to 
upgrade work-related literacy skills at 22 partner sites throughout the Commonwealth of 
Massachusetts. 



Table 1: Year 3 Massachusetts Workplace Literacy Consortium Participants 


Education Partners 


Business/Organization Partners (Location) 


Bristol Community College 


Helix Technology (Mansfield) 
Jostens, Inc. (Attleboro) 
Mason Box (North Attleboro) 
Robbins Company (Attleboro) 


Jewish Vocational Services 


Beth Israel Deaconess/Children's Hospital (Boston) 

Carter Fuller Mental Health Center and AFSCME (Boston) 
C&K Components (Watertown) 

Massachusetts General Hospital (Boston) 

Servolift/Eastern Corporation (Dorchester) 


Labor Education Center at the 
University of Massachusetts, 
Dartmouth 


Lightolier Corporation (Fall River) 


Labor Management Workplace 
Education Program at the 
University of Massachusetts, 
Amherst 


Smith and Wesson Corporation (Springfield) 
University of Massachusetts (Amherst) 


Literacy Volunteers Network 


Holyoke Card and Paper (Springfield) 
Sealed Air Corporation (Holvoke) 


Quinsigamond Community 
College 


Beaumont at the Willows Nursing and Rehabilitation Center 
(Westboro) 

Jewish Healthcare Services (Worcester) 


Service Employees International 
Union/Worker Education Program 


Boston University Medical Center (Boston) 

Harvard Street Neighborhood Health Clinic (Dorchester) 
Jewish Memorial Hospital (Boston) 

Metrowest Medical Center (Framingham) 

St. John of God Hospital (Brighton) 

1 Women’s Educational Industrial Union (Boston) 



The governance structure of the MWLC is based on the Planning and Evaluation Team 
Model. The Consortium Planning and Evaluation Team (CPET) was comprised of the 
program coordinators at each of the seven education provider partners and representatives 
from the DOE. The CPET met monthly to oversee all MWLC activities, identify needs 
among its membership, ensure delivery of appropriate training support and technical 
assistance, and review and address general issues related to providing education services 

8 
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in the workplace. At the site level, the workplace education program at each site was 
organized around a local Planning and Evaluation Team (PET). The local PET was 
comprised of representatives from company management, supervisory staff, 
student/workers, labor unions, and the education provider. The local PETs met regularly 
to oversee the workplace education program at the local level. 

The structure of the MWLC, its educational scope, and its capacity to include large and 
small businesses from diverse sectors of the economy developed out of nine years of 
experience and learning within the DOE about how to best provide workplace literacy 
services to employed workers. It is expected that the lessons learned from the M3YLC 
will provide a strong foundation as workplace initiatives within the Commonwealth move 
forward into their next decade. 



Year 3 Services 

During the third year of the project, the MWLC administered workplace literacy classes 
at twenty-two sites throughout Massachusetts. During Period 5, 572 workers were 
enrolled in 69 classes. During Period 6, 602 students were enrolled in 63 classes. In each 
period, more than half of the workers were enrolled in ESOL programs. A breakdown of 
students by program is presented in Table 2. 



Table 2: Number of Students Enrolled by Program in Period 5 and Period 

6 




Period 5 


Period 6 


ESOL 


389 (68%) 


326 (54.2%) 


ASE/GED 


100(17.5%) 


188 (31.2%) 


ABE 


83 (14.5%) 


80(13.2%) 


Other/Missing Data 


0 (0%) 


8 (1.3%) 


TOTAL 


572 


602 



Given the wide variety of partner sites in the MWLC, there was a wide range in the 
number of classes offered across sites as well as the number of students enrolled in each 
class. Table 3 provides summary descriptive statistics on the enrollment figures and 
course offerings for Periods 5 and 6. 



Table 3: Summary of Enrollment and Course Offerings for Period 5 and Period 6 


average (minimum - maximum) 


Period 5 


Period 6 


Number of classes per site 






Average 


3.5 


2.9 


Median 


2 


2 


Range _____ 


1 - 12 


1 - 13 


Number of students per class 




9.5 


Average 


8.3 


Median 


9 


8 


Range 


1 - 19 


1 - 33 
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Year 3 Evaluation 

The National Workplace Literacy Program requires that an independent external 
evaluation be conducted of each of its projects. In Year 2, the external evaluation was 
structured by four main objectives related to the goals and objectives of the MWLC: 

1 . Determine if the goals of the MWLC are being met. 

2. Determine the level of implementation of the Massachusetts Indicators of 
Quality for Workplace Literacy Programs and the relationship of the Indicators 
to learner outcomes, workplace outcomes and the program partnership. 

3. Determine the relationship between instructional methodologies and 
worker and workplace outcomes. 

4. Develop recommendations for project improvement. 

Based on the results of the Year 2 evaluation and previous evaluations it was determined 
that the Year 3 evaluation would focus on a key aspect of the original MWLC concept: 
the planning and evaluation team model. In particular, the focus of the current evaluation 
is on the organization and operation of planning and evaluation teams at the MWLC and 
partnership site levels. In addition, a final piece of the Year 3 evaluation will be a 
summary of the history of the Indicators of Quality for Workplace Literacy Programs. 
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Section II: Indicators of Quality for Workplace Education Programs 

Overview 



A fundamental element of the Massachusetts Workplace Literacy Consortium has been 
the Indicators of Quality for Workplace Education Programs or Quality Indicators. The 
Quality Indicators were established for six basic components of workplace education 
programs: 



• Partnership and Planning 

• Curriculum 

• Assessment, Evaluation, and Outcomes 

• Support Services 

• Staff 

• Administration. 

The underlying assumption of the Quality Indicators is that they describe the conditions 
which are necessary for a program to deliver effective services. Work on the Quality 
Indicators predates the MWLC and will continue after the conclusion of the current 
project. In this section we present a summary of the Quality Indicator project to date and 
plans for the future. 



Background 



Initial Development 

Drafts of Quality Indicator definitions were initially developed during previous funding 
cycles of the Massachusetts Workplace Literacy Program. They were developed by 
educators brought together through the Massachusetts Workplace Education Initiative. 

In March, 1994, the MWLC’s funding application for Wave VI of the National 
Workplace Literacy Program outlined a central role for the Quality Indicators in the 
program evaluation. The MWLC evaluation provided the opportunity to test the 
assumption of the relationship between the Quality Indicators and the effectiveness ot a 
program. A key component of the evaluation was to examine whether the presence or 
absence of the Indicators correlated with desired outcomes at three levels: 

1 . workers’ educational gains; 

2. productivity gains or improvements in quality of services, and 

3. the quality of the business-union/education provider partnership. 
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Creating an Evaluation Tool 

Before the Quality Indicators could be used in an evaluation, it was necessary to move 
them from initial definitions to a valid and reliable measurement tool. The creation of a 
set of evaluation instruments was a major focus of the initial year of the MWLC. 

During the first year of the program, the external evaluators designed and pilot-tested 
instruments which fully operationalized the Quality Indicators. The evaluators assigned a 
score to each indicator on a 6-point Likert-type scale on which the indicator was rated. In 
making the ratings, evaluators and coordinators considered the dimensions of extent of 
implementation and quality of implementation. Evaluators developed instruments for 
both external and self evaluations. The external ratings were made on the basis of a 
central interview protocol called Indicators PLUS. In addition, shorter versions of the 
Indicators PLUS were designed to be completed by business partners, union 
representatives, teachers, and workers. Finally, a self-evaluation tool was developed for 
coordinators to provide ratings of their own programs. 

Evaluation Results 

The evaluation conducted during the second year of the program showed that the sites and 
partners have implemented the Quality Indicators at a very high level. The average 
ratings on 18 of the 25 indicators were above 5 on the 6-point scale where 6 was the 
highest level of implementation. The strongest areas of implementation were Staff, 
Curriculum, and Administration; the weakest was the area of Assessment, Evaluation, 
and Outcomes. Although the overall level of implementation was quite high, analyses did 
reveal some differences that may be due to type of industry or education provider. More 
research on these factors should be conducted in the future. 

The relationship between the implementation of the Quality Indicators and outcomes was 
not clear from the evaluation. In large part, this was due to the lack of good data 
available on outcomes. Learner outcome data were available in the form of self-reported 
learning gains scores. Other outcome data was scant at best, a by-product of the 
unexpected dearth of NWLIS data. Also, the lack of data is consistent with the relatively 
low level of success at implementing the Quality Indicators in the area of Assessment. 
Evaluation, and Outcomes. 

The Quality Indicators have many benefits when used as a guide for program 
development. However, future research on relationships between implementation of the 
Quality Indicators and outcomes will be necessary to validate the Quality Indicators as an 
accountability measure for workplace education programs. The resources and time 
needed to continue the proper study of these relationships, however, are quite extensive. 
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Performance Standards 

One method of improving the usefulness of the Quality Indicators is to link them to 
clearly defined performance standards. Performance standards can guide programs in 
their development and also serve as benchmarks for developing a scoring system for the 
Quality Indicators. In the third year of the project, efforts were initiated to develop a 
system of program performance standards based on the Quality Indicators. 

Current Status 

The context in which the Quality Indicators were developed was a spirit of collaboration 
between public and private sources to create model workplace literacy programs. The 
emphasis was on education and the focus of the Quality Indicators was on excellence in 
designing and delivering the ideal program. 

The Quality Indicators continue to evolve as 

a) the context expands from public/private collaboration and funding to private 
funding; and 

b) participants have gained three years of experience in striving for the 
institutionalization of their programs 

New issues are emerging and the applications for the indicators are expanding. 

One outcome of the effort to institutionalize programs over the past year is the finding 
that institutionalization is not necessarily a quality issue. In many cases, at least at the 
first cut, the decision whether to institutionalize is a straightforward economic issue: “Is 
there money available in the budget to continue the program?” 

A second finding based on experience is that there are many factors which affect the 
quality of a program over which the education provider has little, if any control. Creating 
standards to hold only education providers accountable to deliver an ideal program has 
limited effectiveness. It is necessary to define standards for all stakeholders that address 
the expectations of the entire program. Such standards would address questions about 
what is required of the company and what is required of the funding agency as well as 
what is required of the education provider. 

The central questions that program staff had to answer were "What purpose can the 
Quality Indicators serve?” and “Who is their audience?” 

The answer to the first question is that the Quality Indicators should be a resource for all 
stakeholders regardless of funding source, company size, type of business, or partnership 
configuration. The challenge is to create a set of indicators that reflect universal quality 
factors that is flexible but still useful. As the final year of the current project was being 
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completed, the quality indicators and supplemental documents were being revised to 
meet this challenge. 

The answer arrived at to the second question is that there are multiple audiences for the 
Quality Indicators. Employers and unions who will be making decisions about whether to 
begin or continue a workplace education program are one type of audience. Agencies 
making funding decisions are another audience. Finally, programs who wish to conduct a 
self-evaluation are another important audience. 

As funding for the MWLC ends, the Quality Indicators and related documents have been 
turned over to the Massachusetts Workplace Education Committee (MWEC) formerly the 
advisory committee to the MWLC. They are reviewing the documents again and will 
continue with further research, development, and implementation. The latest draft 
version of the workplace indicators/standards are included in Appendix A. 
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Section III: Consortium Planning and Evaluation Team 

Overview 

This portion of the evaluation focuses on the functioning of the Consortium Planning and 
Evaluation Team (CPET). Previous evaluations raised questions about the CPET as a 
governing body, the role/authority of the program coordinators, and the role/authority of 
the DOE. Additional questions about the advantages and disadvantages of the 
consortium model for administering workplace education programs are also addressed. 

The information provided here is intended to supplement information obtained in prior 
evaluations. It is based primarily on interviews conducted with the seven program 
coordinators. 



Methodology 



Interview 

0 

The primary method for collecting additional information regarding the functioning of the 
CPET was an individual interview with each of the program coordinators from the seven 
education providers. Two formats were used to conduct the interviews. Five 
coordinators were interviewed as part of site visits to PET meetings. Two coordinators 
whose programs were not part of the PET site visit process were interviewed via 
telephone. The interview was administered in approximately 30 minutes. 

The loosely structured interview focused on areas related to the functioning of the CPET: 

• The role of the CPET and DOE in administering the MWLC; 

• The roles of the DOE and program coordinators in the CPET: and 

• The sense of belonging to a consortium. 

Two initial questions in each area were scripted. Follow-up questions were asked based 
on responses to the initial questions. Additional questions were also included to 
supplement the information gathered about the PETs. A copy of the interview protocol is 
provided in Appendix B. 

Observation 

Observational information about the functioning of the CPET was collected by attending 
the June 3, 1997 CPET meeting in Springfield, Massachusetts. The evaluator paid 
particular attention to 

• how the meeting was conducted 

• interactions between program coordinators and DOE personnel 

• interactions among program coordinators. 
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The meeting lasted approximately 4 hours. It was attended by coordinators from three of 
the seven programs, a representative from a fourth program, and three representatives of 
the department of education. 

Archival Records 

To provide background and supplementary information the evaluator reviewed archival 
materials related to the functioning of the CPET. These materials included minutes of 
CPET meetings, surveys administered to CPET members, materials presented to the 
CPET, and information collected in previous evaluations. 

Background 

The CPET was established to mirror the team approach of the PET model at the 
partnership level. The CPET includes the project staff from the DOE and each of the 
seven program coordinators. The original application for funding for the Massachusetts 
Workplace Literacy^ Consortium (March, 1994) provides the following description of the 
CPET: 



All Consortium activities are overseen by the Consortium Planning and 
Evaluation Team (Consortium P.E.T.). This united body of partnership 
coordinators and a state-hired Consortium staff (full-time Coordinator and 
half time Assistant Coordinator) works to identify needs among its 
membership, ensures delivery of appropriate training support and technical 
assistance, and, like the partnership P.E.T.s, evaluates its activities in two 
written reports during the course of the three years’ funding, (p.48) 

Questions/Discussion 

• Did the CPET act as a governing body? 

• Was there a sense of belonging to a consortium? 

• What were the advantages/disadvantages of the consortium model? 



Question 1: Did the Consortium PET act as a governing body? 



Working Toward Quality >, the Year 2 evaluation of the MWLC addressed the issue of 
“ The Consortium PET as Governing Body. ” It is clear from the discussion in that report 
that during the implementation of the project there was some level of disagreement about 
the responsibilities of CPET members and the role of the CPET in governing the MWLC. 
A review of minutes of CPET meetings from the period March, 1995 through November, 
1996 revealed that the function of the CPET and its range and limits of authority to 
govern were two issues that dominated the agenda during that time period. 
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Based on the materials reviewed and interviews conducted for the current evaluation, it 
can be concluded that the CPET did not govern the MWLC. As described by the program 
coordinators, the CPET served two roles in the governance of the MWLC. First, the 
CPET served in an advisory capacity to the DOE. Second, the CPET members (program 
coordinators) served as managers carrying out the directives of the DOE and fulfilling 
reporting requirements for the program. 

It was clear from the interviews, however, that several of the coordi uators did not 
understand or accept this role at the beginning of the project. These coordinators believed 
that the governance of the consortium would be shared by the DOE and program 
coordinators through the CPET. Over time, all coordinators understood their role in the 
project although they did not all fully accept it. An example from the final CPET meeting 
in June, 1997 illustrates this point. At that meeting, the Consortium Coordinator, a DOE 
representative, presented the coordinators with a list of options for using remaining funds 
available to the project. The list had been compiled by the DOE prior to the CPET 
meeting and some potential options had been eliminated from consideiation. 

Interactions at the meeting and conversations between the evaluator and coordinators 
after the meeting revealed that some coordinators were not pleased with their limited role 
in the decision-making process. 

Question 2: Was there a sense of belonging to a consortium? 

The program coordinators had mixed responses to the question of whether there was a 
sense of belonging to a consortium in the project. The overall impression that emerged 
from responses to this question was that coordinators felt that the operating model that 
emerged during the project did not live up to their expectations of a consortium, but did 
have many benefits over working alone. They did feel that they were part of a team 
working toward a single goal. The coordinators provided strong positive feedback about 
the role of the CPET in accomplishing tasks such as { 

l 

• establishing mechanisms for support and sharing of resources, e xpertise. and effective 
operational models for all partnerships, and 

• establishing a curriculum focus group to develop curriculum guidelines and 
frameworks to link curricula of different projects. 

Overall, the lack of a sense of a consortium was attributed to two related factors 
concerning the management of the project by the DOE: 

1 . lack of control over spending decisions; and 

2. more DOE control than anticipated. 

Throughout the project, there was a sense that many decisions wer? being made outside 
of the CPET and delivered to the CPET by the Consortium Coordinators. Adding to the 
feeling of a lack of control were the many reporting requirements ot the Massachusetts 
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DOE and the US DOE. The failure of the computerized system for collecting data 
required for the US DOE added to the coordinators’ sense of frustration. 

Throughout the interviews, coordinators separated their opinions on the control exercised 
by the DOE and the support provided by the Consortium Coordinato rs (representatives of 
the DOE). As will be discussed in more detail in response to the next question, 
coordinators were generally positive about the quantity and quality of efforts of the 
Consortium Coordinators. ; 

In joining the Massachusetts Workplace Literacy Consortium, man}' program 
coordinators felt that their programs were entering into a full partnership or alliance with 
the MA DOE and the other education providers. In the end, the con census was that the 
relationship was something less than a full partnership. However, this did not lessen 
many of the benefits derived from the consortium model. 

Question 3: What were the advantages/disadvantages of the consortium model? 

The consortium model was designed to enhance the literacy programs in each of its 
partner sites. Through features such as cooperative planning efforts, comprehensive 
programs of staff training and resource sharing, technical assistance and instructional 
support the MWLC would offer partners a more cost effective method for delivering 
services. Further, the pooling of resources in the MWLC would allow many small 
businesses to offer services that otherwise would be beyond their means. The application 
for funding for the Massachusetts Workplace Literacy Consortium included a list of 
advantages to education providers and business partners of the consortium model (Table 
4). 



Table 4: Advantages of the Consortium Model 

The consortium model: J 

• builds upon nine years of Massachusetts Workplace Education successes and 
mistakes 

• is most cost effective 

• coordinates and distributes available resources 

• avoids duplication of effort and services, and 

• greatly facilitates communication and dissemination of in ormation among 

partnerships sharing similar concerns. 

The model provides clear strategies to: 

• address the needs of small business within a larger framework 

• build the capacity of individual partnerships 

• draw upon the wealth of expertise and resources within ti e state, and 

• produce and disseminate materials in the field that are specific to industries 
critical to the state’s economic recovery and renewed competitiveness. 



There was general agreement among the program coordinators that :he MWLP did 
provide the benefits listed in Table 4. Asked during the interview to discuss the 
advantages of belonging to the consortium at least one coordinator mentioned each of the 
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nine advantages listed and most of the advantages were mentioned by several of the 
coordinators. 

There were three themes that dominated the coordinators’ discussion of the advantages of 
the consortium model. In one form or another, the coordinators’ responses focused on 

1 . the cost effectiveness of the model, 

2. the ability to share resources and draw on the resources of the state, and 

3. the opportunities for communication with colleagues. 

Coordinators also mentioned several specific products or services that were 
developed/disseminated during the course of the MWLC. In particular, several 
coordinators praised the consortium’s efforts in the area of distance education. One 
project developed was the use of televised mini-courses which were offered in the fall of 
1995 and 1996. Another project was the development and implementation of a computer- 
assisted instruction package with a focus on basic literacy skills in the health care 
industry. Finally, the MWLC also explored options for the use of video conferencing to 
effectively deliver material to more students. Apparently, some of these efforts were 
more successful than others. However, even when discussing an offering that was not 
very successful or not as relevant to their particular program, coordinators consistently 
presented the effort in a very positive manner. 

A final observation about coordinators’ discussion of the advantages of the consortium 
model concerns the role of the MA DOE. In contrast to their discussion of the role of the 
DOE in the governance of the MWLC, coordinators were quite positive about the role of 
the DOE in facilitating a) the development and dissemination of materials and services, 
and b) the opportunities for communication among the MWLC partners. Separating out 
the role of the DOE as manager of the MWLC, the program coordinators were positive 
about the efforts of the DOE, and in particular the Consortium coordinators, to provide 
the support needed to accomplish the goals of the MWLC. 

Coordinators did not provide any specific examples of disadvantages of the consortium 
model. That is, even those who were especially dissatisfied with the governance of the 
MWLC, did not indicate that their program suffered at all from membership in the 
MWLC. The only negative concern expressed by coordinators was the time required to 
travel to and from the monthly CPET meetings. 
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Section IV: Planning and Evaluation Team 

Overview 

The central focus of the Year 3 evaluation was the functioning and organization of the 
local Planning and Evaluation Teams (PETs). The PETs were designed to mirror the 
partnership model that characterized the MWLC. The PETs were established as the local 
governing body for the workplace education program'within each organization. They 
included representatives from all relevant stakeholders including the education provider, 
teachers, company management, unions, and student/workers. This evaluation addressed 
issues such as 

• participation and roles of members of the organization: 

• problems/barriers to full participation in the PET; 

• obstacles faced by the PET; 

• types of decision-making process/style; 

• meeting environment; and 

• role of the PET in the institutionalization of the workplace literacy program. 

Methodology 



Data Collection 

A variety of methods were used to collect data concerning the PETs. Most of the data 
were collected during site' visits to a sample of 10 PETs. Each site visit provided a) 
observational data, b) a participation chart, c) a group interview, and d) individual surveys 
completed by PET members. In addition to the site visits, additional information 
concerning the PETs was collected as part of the individual interviews with the program 
coordinators conducted for the CPET evaluation. The program coordinator interview 
process is described fully in the CPET section of this report. This section will focus on 
the methods used to collect data during the site visit. 

Site Visit Sample 

The sample used for this portion of the evaluation was PETs from 10 of the 22 
partnership sites. The sites selected represented a cross section of the MWLC 
partnerships based on factors such as type of industry, location, organization (union/non- 
union), and company size. The breakdown of the sample selected on each of these factors 
is presented in Table 5. A complete list of the sites visited is presented in Appendix C. 
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Table 5: Characteristics of Sample Sites 

Industry: 

Healthcare - 5 
Manufacturing - 4 

Education - 1 

Location: 

Greater Boston - 5 
Central Massachusetts - 2 
Southeastern Massachusetts - 1 

Western Massachusetts - 2 

Organization: 

Union - 3 

Non-union - 7 

Size: 

Mid-size - 5 

Large - 5 



Site Visit Process 

The evaluator conducted the site visits between April, 1997 and July, 1997. The original 
plan called for the following process to be conducted at each site visit: 

® PET participation chart completed 

• observation of the PET meeting 

• administration of the group interview with PET members 

• distribution of the individual surveys. 

• collection of ancillary materials such as meeting agenda and reports 
distributed at the meeting. 

This format was followed at six of the ten sites. At the remaining four sites, the 
evaluation was the only item on the PET meeting agenda which eliminated the 
opportunity to observe a full PET meeting. Full PET meetings lasted approximately one 
hour. Evaluation only visits took approximately 30 minutes. 

PET Participation Chart 

A PET Participation Chart (see Appendix D) was completed at the beginning of each 
meeting to collect the following information from each person attending the meeting: 



• Name 

• Position (e.g., education provider, management, worker/student) 

• Current PET membership status 

• Time served on PET 

• Number of PET meetings attended during the past 12 months. 
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The program coordinator was also asked to complete the chart for current PET members 
who were unable to attend the meeting. The chart provided information on representation 
on the PET and turnover of PET members. 

Group Interview 

The group interview was the primary method for collecting data during the site visit. The 
interview was conducted by the evaluator using a survey/interview protocol which was 
distributed to all PET members. PET members were given the option of providing 
additional information in written form and returning the interview protocol with their 
individual survey. This option was included to accommodate PET members who may not 
have been comfortable responding in the group setting. 

The interview protocol was developed with input from the evaluator, the Consortium 
Coordinator, and the evaluation unit of the DOE. A draft of the survey was distributed to 
program coordinators for their review and comment. The protocol contained 1 1 
questions concerning 

• participation in the PET; 

• appropriateness of the PET model; 

• success/obstacles to the workplace education program; 

• support structure for the PET ; 

• advantages/disadvantages of the PET model; and 

• institutionalization. 

All but one of the questions were either open-response or a combination of forced-choice 
and open-response. The first question was a forced choice question concerning 
participation in PET governance tasks. A copy of the interview protocol is provided in 
Appendix D. 

Individual Survey 

An individual survey was used to supplement information collected during the group 
interview. The survey addressed many of the same issues as the group interview, but was 
designed to elicit opinions that PET members may not have been comfortable expressing 
in front of their colleagues. PET members were provided with a postage-paid, addressed 
envelope to return the survey. 

The survey contained eight questions. Two of the questions were forced-choice 
questions. The remaining questions were open-response or a combination of forced- 
choice and open response. Like the group interview, the individual survey was developed 
with input from the evaluator, the Consortium Coordinator, and the evaluation unit of the 
DOE. A copy of the individual survey is provided in Appendix D. 
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Completed surveys were received from 28 PET members. At least one survey was 
returned from each PET visited. Four surveys each were returned from three of the PET 
sites. One of the returned surveys was completed by a current student and one was 
completed by a former student (currently a union representative). A distribution of the 
persons returning surveys by position is provided in Figure 1 . 



Figure 1: Respondents to Individual Survey by Position 
Number of surveys returned 
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Observation Form 

The evaluator completed an observation form at each PET site visit. The form was used 
to collect information on the layout of the meeting room, seating arrangements, 
participation, and organization of the meeting. A copy of the form is provided in 
Appendix D. 



Questions/Discussion 

Question 1: Is there participation on the PET by all relevant stakeholders? 

To properly answer this question, it is necessary to separate it into two parts: 

1 . Is there participation on the PET by all other relevant stakeholders? 

2. Is there participation on the PET by students/workers? 
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It was clear from both observations of the PET meetings and responses from PET 
members that student participation on the PET was a distinct issue from participation by 
all other members. Issues such as language barriers, position within the company, and 
length of time on the PET present unique challenges to student participation. Therefore, 
student participation on the PETs will be addressed in response to Question 2. In this 
section, the focus will be on the participation of other stakeholders. 

There is no evidence of explicit or implicit barriers on the participation of any members 
of the PET. Observations of PET meetings revealed a high level of interaction and 
participation by all PET members. In general, informal interactions among all PET 
members began prior to the start of the meeting and continued throughout the meeting. 
Members appeared to be comfortable voicing their opinions. Survey responses also 
indicated that all members were able to fully participate in PET discussions and decision- 
making. 

A more serious threat to full participation, however, may be certain groups opting out of 
the PET process. At two sites, lack of consistent participation by company management 
(other than the human resources director) was identified as a problem for the PET. At 
another site, under-representation of supervisory staff was a concern. 

Although participation by all members was considered important, PET members placed 
particular importance on the participation of supervisory level staff. Gaining input and 
buy-in from workers' direct supervisors was identified as a critical factor in determining 
the success of the PET and the workplace education program. One problem identified at 
three of the large company sites was the difficulty of gaining buy-in from representatives 
from all departments affected by the workplace education program. Their PETs had been 
designed to include one or two staff members from the supervisory level, but there were 
five or more departments with students participating in the program. They did not 
recommend increasing the size of the PET to include all supervisors, but emphasized that 
measures had to be taken to ensure that input was received from all supervisors. 

Question 2: Is there participation on the PET bv students/workers? 

According to coordinators, concern about student participation on the PETs has been a 
major issue since the beginning of the project. There was a fear that although student 
input was critical, obtaining student participation could be the weak link in the PET 
model. Coordinators anticipated that the greatest obstacles to student participation would 
be students’ 

• difficulties in expressing themselves in English 

• reluctance to speak in front of their supervisors and other company 
management. 

Observations of the PET meetings and survey responses indicate that lack of student 
participation was a problem for some PETs. One-quarter of the survey respondents 
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responded “No” to the question ‘‘Do students have a full voice in the activities of your 
PET?’’. The most common explanation for their response was students’ inability to 
speak English. This was especially true for beginning ESL students who were PET 
members. The reluctance of students to speak in front of management was also 
mentioned on the surveys by a small number of respondents. Observations at the PET 
meetings support the argument that language barriers were a greater problem than 
reluctance to speak. At four PET meetings where the student member was fluent in 
English, he/she actively participated in the discussions. (It should also be noted, 
however, that in only one case was a controversial or confrontational issue discussed at 
the meetings attended by the evaluator. That issue was raised by the student 
representative.) 

Another issue that emerged as a potential barrier to students’ effectiveness on the PET 
was their rate of turnover on the PET. On some PETs there was a high rate of turnover of 
the student representative due to a) students’ completing the program, or b) a planned 
system of rotating students through the PET. In other cases, the same student had served 
on the PET since its inception. As expected, students with greater longevity on the PET 
appeared to be more comfortable and more willing to participate. 

As was the case with supervisors, under-representation of students was mentioned as a 
problem at the larger companies. In many cases, there was a single student representative 
in a program with more than 100 students spread across 10-15 classes. One site 
attempted to solve this problem by forming a “student PET” with members from each of 
the classes. This group would meet shortly before the regular PET meeting and prepare a 
report to be delivered by their representative on the PET. 

PETs have tried a number of approaches to support and encourage student participation. 
The most commonly used approaches involved attempts to make student involvement a 
regular part of the PET meeting. In some cases, the program coordinator made it a point 
to regularly direct questions to the student representative. In other cases, a student report 
was made a regular agenda item. Also, many teachers reported meeting with the student 
representative individually prior to the PET meeting. The distribution of responses to the 
survey question, "'What does your PET do to encourage student participation and input 
into PET discussions and decisions? ” is presented in Figure 2. 
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Figure 2: Efforts Taken to Promote Student Participation in the PET 




Question 3: Is a PET important to the success of a workplace education program? 

There was unanimous agreement that the PET model was critical to the functioning of a 
workplace education program. In the group interview, individual interviews with the 
program coordinators, and responses to individual surveys PET members consistently 
expressed the opinion that it would be very difficult to develop, implement, and 
institutionalize a workplace education program without a PET. PET members 
unanimously responded ‘yes’ to the following questions: 



• If you were starting a new workplace education program, would you have, a Planning 
and Evaluation Team? 

• Would you want to he part of a future Planning and Evaluation Team ? 



PET members provided a variety of reasons for the importance of the PET. The most 
commonly cited reasons were that the PET 

• provides the opportunity to keep all stakeholders informed on a regular basis; 

• presents a forum for open discussion; 

• strengthens the partnership and team approach within the organization; and 

• promotes buy-in from all stakeholders. 
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A complete distribution of stated benefits of the PET model in enhancing the workplace 
education program is provided in Figure 3. 



Figure 3: Benefits of the PET Model 

Number of Responses 
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A great deal of emphasis was placed on the importance of the PET in obtaining buy-in 
from all stakeholders. In particular, participation and buy-in from management and 
supervisory staff was considered a key to success. 

In addition to informing all stakeholders , a significant group of respondents also made it 
a point to mention the importance of the PET in providing feedback to the teacher. 
Teachers were able to better shape/adjust their curriculum, make scheduling changes, and 
understand the organization because of their participation on the PET. 

The only disadvantage of the PET model mentioned by PET members was the time 
commitment necessary to have an effective PET. One suggestion consistently offered as 
a change to the PET model was to have less frequent (but regularly scheduled) meetings 
after the program was established. It should be noted that attendance records indicated 
that most current PET members attend nine or ten meetings per year. 

Another issue mentioned as detrimental to the PET model was the lack of participation by 
key company personnel. In three cases, the PET had a great deal of trouble obtaining 
consistent participation from company management due to a high rates of turnover and 
corporate restructuring. 
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Question 4: How are PET meetings conducted? 

Across industries and types of organization there was a uniformity to the PET meetings 
observed for this evaluation. In all cases the PET meetings were conducted in an 
informal and relaxed manner. Although there was a chairperson conducting each 
meeting, interactions between all members were quite unrestrained. In general, the role 
of the chairperson was to ensure that all items on the agenda were discussed and that a 
date was set for the next meeting. 

o' 

In all but one case, the program coordinator chaired the meeting. Teachers, however, 
were often the focus of most of the discussion: providing class updates, attendance 
reports, and providing/seeking information of other PET members. With the exception of 
some student representatives, all PET members participated at every meeting attended. 
The interactions were all friendly and professional. 

In general, the meetings were held in a comfortable, quiet environment. Most meetings 
were held in a conference room or board room. At the manufacturing companies, the 
meetings were held close to the work floor, but in a quiet room. 

Question 5: Did the PET receive sufficient support? 

PET members were asked to indicate what additional support and assistance their PET 
could have received during the course of the project. Individuals were also asked to rate 
the quality of support provided by the following groups: 

• Department of Education 

• Consortium PET 

• Program Coordinator 

• Other PET members 

• Business/Organization 

• Union 

Overall, in both the group interview and on the individual surveys, members gave high 
ratings to the support provided by the program coordinator, other PET members, the 
organization, and the union. There were specific cases, noted previously, where the 
management has not been very involved in the PET process and this was reflected in the 
ratings given at those sites. 

Support from the DOE was discussed primarily in terms of materials provided or 
produced during the project and conference opportunities. PET members were positive 
about these efforts. The only specific additional support requested of the DOE was more 
of a presence on site. Several PET members indicated that they would have liked to have 
had a DOE representative visit their meetings, classes, or activities two to three times per 
year. 
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The discussion of support from the Consortium PET was quite interesting. PET members 
had virtually no knowledge of the Consortium PET or the consortium. The few 
exceptions were members who had attended state or national conferences. Many PET 
members were aware that their provider was in partnership with other businesses, but 
were unaware of a statewide network. 

Question 6; What role does the PET play in the institutionalization of the program? 

! 

The simple answer to this question is that the PET is helpful but not sufficient to ensure 
that the program is institutionalized. Program coordinators and PET members at all sites 
reported that the decision to continue or institutionalize the workplace education program 
would be based on factors other than the success of the program. In many cases, 
however, they did feel that without the buy-in and teamwork promoted by the PET that 
the organization would not even be considering continuing the program. At one site with 
a strong PET. the continuation of support for the workplace education program was one 
of the first items that labor and management agreed upon during a tense contract 
negotiation. 



At all sites, PET members expressed the opinion that all of their act: vities throughout the 
course of the project were aimed at enhancing the chances for the success of their 
programs and therefore helping to ensure its continuation. Additionally, at four sites the 
PET had been actively involved in preparing reports, budgets, or proposals for the 
continuation of the program beyond the current funding cycle. However, as PET 
members at two sites mentioned, it must also be remembered that many PET members 
(e.g., program coordinators, teachers, students, unions) may have high personal stakes 
associated with the continuation of the program. This could confou nd an interpretation 
of the actions of the PET toward ensuring that the program continues. 



29 



Massachusetts Workplace Literacy Consortium Year 3 Evaluation, October, 1997 Page 22 



Appendix A 



30 




Massachusetts Workplace Literacy Consortium Year 3 Evaluation, October, 1997 



Program Standards and Quality Indicators 
for Workplace Education 



Massachusetts Department of Education 
October, 1997 



3.1 



Introduction 



The Development of Standards and Indicators 

The Draft Standards and Quality Indicators for Workplace Education Programs define the 
conditions under which effective education programs can best develop, and identify outcomes 
and impacts expected of those programs. The current iteration is the result of many rounds of 
input from representative groups of experienced workplace educators, employers, union 
representatives, policy-makers, academics and evaluators. Their on-going review has enabled the 
Standards/Indicators to become more specific, while retaining their relevance to programs that 
may be diverse by size, funding source, industry, union involvement, and type of partnership 
(workplace and education provider, consortium, in-house labor-management program, etc.). 

The current Draft Standards and Indicators are intended to make the collective knowledse and 
experience of the field available to guide program start-up and development, on-going self- 
evaluation, and funding decisions. 



Underlying Assumptions 

Embedded in the Standards/Indicators are some assumptions that should be made explicit. The 
first is that learning to read, write, do math, or speak a new language requires a significant 
commitment of time. Unlike skill enhancement workshops that may be effective after a few 
hours, basic education for those who are learning for the first time takes hundreds of hours (see 
Adult and Community Learning Services (ACLS)/DOE sendee delivery chart below). There are 
no long-lasting shortcuts. The workplace context demands thoughtful consideration of how to 
satisfy work coverage and production needs while addressing educational goals realistically. 

Second, Standards/Indicators presume the availability of technical assistance and support for 
developing programs. Since many of the most promising practices in workplace education have 
short histories and limited documentation, there remains a need for program support and staff 
development. Specifically, programs may need assistance in areas such as workplace needs 
analyses, authentic assessment of program outcomes and impacts, and curriculum development. 



Finally, the standards acknowledge that workplace education is only effective when it responds 
to both the self-defined needs of worker-students and employers, which may differ. Workers may 
piefer to practice their skills using community and family topics, which may not immediately 
appear to be work-related. However, high-interest lessons will expedite the acquisition of basic 
skills, which will then transfer to the workplace. This transfer is the responsibility of the 
educators, who relate the coursework to workplace applications, and of the other partners, who 
create workplace opportunities for students to practice and demonstrate their learning. 



The Document 



The Standards/Indicators document is organized according to three aspects of a workplace 
program - governance, staffing, and student services. Each of the three opens with a cover sheet 
outlining the standards (expectations of performance) and related quality indicators (evidence of 
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achievement) of excellence. Attached to this is a chart which repeats the standards and lists not 
only the quality indicators of excellent programs, but also quality indicators of basic and 
developing programs. These descriptions incorporate what has been learned by seven years of 
public/private collaborations and can be helpful in self-evaluating a program’s strengths and 
weaknesses. 

Definitions of Terms 

The follow terms are used throughout the Standards/Indicators document and are intended to 
mean: 

stakeholders: all parties that have an interest in workplace education including, but not limited to 
management, supervisors, employees, union, education coordinator/provider, and teachers. The 
governing team (GT) is made up of alj stakeholders. 

workplace needs analysis (WNA) : a process of assessing the needs of an organization in order to 
identify which needs can be appropriately addressed by a workplace education program. In this 
process, a representative sample of all employees are interviewed. 

adult basi c education (ABE) curriculum frameworks : field-generated guidelines for applying 
current adult learning theory to the classroom. 



ACLS GUIDELINES FOR CLASS SIZE AND HOURS 



Service Type 


Class Size 


Estimated Hours 
Needed to 
Complete 
Category 


Minimal 

Contact 

Hours 

Needed 


Hours of Instruction 
Per Week 


Range 


Preferred 


Range 


Optimal 


ABE/Literocy 

(0-5.9) 


5-20 


5-10 


650-900 


200 


5-20 


8-15 


Pre-ASE 

(6-8.9) 


5-20 


7-15 


150-300 


65 


5-20 


8-1 5 


ASE 

(9-12) 


5-20 


9-15 


120-300 


20 


5-20 


8-15 


ESOL 
(SPL 0^) 


5-20 


1 1-20 


450-700 


125-150 


5-20 


8-15 


ESOL 
(SPL 5-6) 


5-20 


1 1-20 


300-450 


100 


5-20 


8-15 


ESOL 

(SPL 7-10) 


5-20 


11-20 


200-300 


50 


5-20 


8-15 



o 
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Governance 



Standard #1: There is a formal, well-attended partnership/goveming team (GT) established among all 
stakeholders (including management, supervisors, employees, education provider, teachers, and union 
where present) which oversees the design, development, implementation, and evaluation of the program. 

Indicators: - All stakeholders can access meetings and participate consistently on the GT; policies are 

in place which remove barriers to employee participation. All stakeholder representatives 
participate in more than 75% of governance activities 

- GT has established legitimacy and the support of all stakeholders 

- There are regular, facilitated meetings of the GT and on-going communication about 
program and governance activities 

- GT has full authority to develop and implement program goals and policies, and to direct 
practices 

- There are regular (class) meetings for students to articulate needs and prepare their repre- 
sentatives to participate in GT meetings. There is a student representative from every class 

Standard #2: There are policies and procedures in place to ensure that the GT functions effectively, 
makes informed decisions, and maintains a supportive learning environment. 

Indicators: - There are standardized procedures for governing and a clear decision-making process. 

Roles and responsibilities are clear; policies and procedures written 

- GT is informed by the stakeholders’ workplace needs and strengths, resources, 
demographics, principles of adult education, and trends in the workplace/industry 

- GT defines clearly-articulated goals and expectations for the program which are aligned 
with the long- and short-term goals of all stakeholders, and which are used to develop an 
action plan. There is a process in place to integrate GT goals and class-defined goals 

- GT uses evaluation data that is relevant to program goals, reliable, accessible, and 
regularly updated to evaluate and revise program goals and practices. GT responds 
flexibly to changing circumstances 

- There is a full workplace needs assessment (WNA) of contextual, organizational, and 
learner needs, strengths, and goals 



Standard #3: Resources and policies support daily program operations as well as the long-term 
institutionalization of workplace education into the workplace culture and union contract. 



Indicators: - Resources are adequate to support a full range of student services, competitive salaries 

for staff, staff training, workplace needs analyses, program evaluation, and governance 
activities 
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Student Services 



Standard #1: There axe learning opportunities and support services designed to meet the needs and goals 
of adults at the workplace, and an educational ladder linked to a full sequence of educational services in 
the community. 

Indicators: - Classes and other learning activities are offered at the appropriate level, time, and place 

(with transportation if off-site) to meet the needs of all students. Classes and other 
learning activities are scheduled consistently and in long enough duration for learning to 
occur. Delivery may be in the form of classes, 1-1 tutoring, on-the-job support, and other 
applicable models 

- There is a non-discriminatory process for selecting and placing interested students. All 
those who want services can access instructional opportunities/support services 
consistently because the program has policies and incentives that support participation 
without jeopardizing coverage, production or service delivery, and family responsibilities. 
Students have access to program offerings until program goals are met 

- Students receive 100% paid release time and other incentives for participation, 
completion, and/or achievement (or compensation as stipulated in union contract) 

- There are no employment-related repercussions for participation, non-participation or 
lack of progress. Policies protect the confidentiality of student records 

- Support services (educational counseling, transportation to off-site services, peer 
advocates, daycare, etc) are in place on-site, as needed, or readily available and easily 
accessible. All students receive information and counseling support regarding 
opportunities for further education and training 

- Facilities and services are handicapped-accessible and provide for a safe and comfortable 
learning and working environment for students and staff. Program ability to provide 
accommodations exists. All staff are trained and integrate support services into daily 
program operations 



Standard #2: The program has a curriculum that flexibly responds to the changing needs, strengths, and 
goals of students and the GT. 

Indicators: - Program uses a systematic process for regularly assessing learning needs, strengths, and 

goals and revising the curriculum accordingly 

- Student learning needs are determined by a variety of tools, including individual self- 
assessment and group goal-settimg 

- Lessons consistently incorporate the needs, strengths, learning stvles. experiences, and 
perspectives of the students 

- Teacher adapts materials so that they are relevant to the context (the students’ lives, 
union, and workplace). Teacher uses a variety of materials and activities in an adult- 
appropriate manner and applies principles of adult learning as articulated in the ABE 
curriculum frameworks 

- Classes focus on the development of skills, knowledge and abilities that are transferable 
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to various contexts (within and outside of work) and help students make these 
transferences. Lessons include but are not limited to workplace-related content 

- Curriculum encourages critical thinking skills so that students can improve their lives 
and workplace, and can apply new skills and knowledge to problematic situations. 



Standard #3: There is demonstrable progress, according to all stakeholders, toward program goals. 

Indicators: - There is demonstrable evidence of substantial progress (toward anticipated and 

unanticipated goals) for all stakeholders. 

- There is recognition that changing conditions and/or goals affect performance and 

impact. 

- Program uses a variety of evaluation tools to document a comprehensive, “authentic” 
view of student progress and satisfaction. 

- Policies and practices are in place to ensure that workers are given opportunities to 

practice and demonstrate new abilities on the job. 

- At least 75% of the students enrolled in the program participate consistently in planned 

instructional activities. 

* At least 75% of the students achieve the learning standards or objectives of instruction. 

- Those who do not achieve the standards or objects are evaluated and receive guidance 
for further education or services 

- All students can apply what they have learned at work, in the union, at home, and in the 
community. 

- Students who pursue further education and training are tracked at least once after leavin 
the program. 



Studept Services 

Standard #1: There are learning opportunities and support services designed to meet the needs and goals of adults at the workplace, and an educational ladder 
linked to a full sequence of educational services in the community. 
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Standard #3: There is demonstrable progress, according to all stakeholders, toward program goals. 
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Staffing 



Standard #1: Staff is competent to teach the relevant subject matter to a diverse population of adults in the 
workplace. 



Indicators: 



- Staff has experience teaching adults in workplace settings, and has experience working 

with multi-ethnic populations , 

- Staff has experience and skill in teaching the relevant si 1 bject matter: reading and 
writing, oral communication, math, etc. 

- Staff is skilled in creating original curricula tailored to each worksite and responsive to 
the goals of each group of students 

- Staff is skilled in assessing and evaluating learners’ skills and progress and in teaching 
students how to evaluate their own learning 

- Staff is skilled in working within a business, union, or 1 ibor/management context. Staff 
has some expertise in gathering qualitative and quantitative data for diverse stakeholders 



Standard #2: Staff is compensated at competitive salary levels for teaching, curriculum development, 
professional development, student assessment, and program development. 

Indicators: - Staff is a mix of part-time and full-time, with full benefiis 

- Stalf is paid for contact hours and at least 2 hours of prep time, 2 hours of curriculum 
development time, 1 hour of assessment and documentation time, and I hour of planning/ 
adminstrative time for each four contact hours worked (6:4). At least 2.5% of their time is 
allocated to training and development. At least 4.5% of th eir time is allocated to program 
planning, recruitment, evaluation, and improvement. This includes attending company- 
sponsored training, communicating with company personnel, time on the floor 

- All stakeholders participate in offering a coinprehensive.staff orientation to the 

workplace, its culture and dynamics • 

- Staff development opportunities include specialized workplace education trainings 
informed by trends in the industry', the workplace, the labor movement, and the field of 
adult education, and in-house company training in which students/employees attend. This 
will increase staff s understanding of business issues, needs, and processes 



Standard #3: Staffing is adequate to cover the tasks and functions needed to support students through the 
program. “ 



Indicators: 



- Stall is skilled in educational counseling and uses referral network 

- Program involves all workplace personnel and orients them to principles of adult 
learning 

- There are trained support service workers on staff (counselors, day care workers, etc.) 

- Staff turnover is low 



Staffing 

Standard #1 : Staff is competent to teach the relevant subject matter to a diverse population of adults in the workplace. 




<L> 

O 

X 

PJ 



*u 

c 



60 



o V'- 

Vi C 

o .2 

J a 

a 2 
a. 

Jr O 
O Cu 

> CJ 

.5 *c 
w S 
+* o 
3 .JL 
~0 «=S 

03. • 3 

60 E 



JC — 

£ 5 
2 60 
* ’k Cl 



CL O 
x 2 
° o 

5 O 
, a. 

x 

_a 

£ a 

. 4= 



jO 

3 



3 

« oj 
> 3 

o — 



— <3 

CO ~ 

c c 



p § 



33 

o 



o 

V- 

60 



3 CJ 
O 



= S 

•- c 



Z ■£ 

c a 

« § 
o 

— 03 

CO > 
o 



- £ 

O 3 O 



t: o 
o -X 

CL 5 



E O 



J3 

O 

C3 

O 



2 ® 
5/2 60 
33 c 
3 ‘3 



Q. ^ 
X -3 
O -3 
wi E 
03 O 
J3 

O 

on 2 



ri 

O 

c*_ 


'5 


C 


V. 

3 


0 


0 

> 


O 

iyj 


3 


o 


_C 


Vi 

t/5 


■5 


3 

o 


> 

o 


J3 

vc 


3 




u. 

O 


to 


33 

3 


3 


"3 


cc 


3 


o 

c 


60 

E 

Ji 


_o 

*3 

VC 


P> 


0 ^ 

x 


O 

> 


o 


60 


-_2 


0 




> 


O 




0 


3 


3 


V. 


VC 


’ ~ 


> 


3 


‘3 


3 


3 


O 




0 


c 




. 










CL 


.3 


O 


,= 


3 


3 



v: 

V) VS 



00 O 

t > 



p- ° 

£3 DO 

on 



O 2 
CL -3 



I E 3 

^ - O 

■/ X > 



J « 



60 

E 

'5* 

jD 

> 

o 

Q 



33 

o 

3 



• 3 O 

O g 

g 3 
■- 22 
^*5) 

x IE 
2 * 
3 c2 

1?° 

c/T 

~ too 
os c 
o • — 



"3 ■£ 

a S 
£ 5 
2 - 
to 

3 33 

¥ J 



3 

o 



CL 

o 



~3 

O 



2 33 
60 — G 
,£u ^ 
c O ^ 

2 E 



-v w 

»■* >o — 



0 — 03 : 



o 3 
33 ‘ v 
3 O 



E O 3 

O vs 



5 s - 

w ~x 3 



c 

o 



o 

"3 

3 



vs O 
C3 CJ 
43 £2 

£: a. 

2 -a 

00 O 

I ? 



60 3 
3 33 
• ~ o 

o — 

VS 3 

o -P 

CJ « 

J2 c 

-& p 



c2 ° 

o i2 

a. P 

c C3 
* 2 
2 **■' 
3 c 
a. — 

'3 </i 

.2 o 



o 

a. 

x 



S E 
■ Ci. 3 

1 I 

00 CL 

I o 



o 

S) "• 

3 ”2 

CJ H 



11 

II 

Q w 
o o 0> 

> ^ O 

0 ^ JZ 
^ rt - 

1 E g 

3 -2 •§ 



3 2 

c. — 
E 
o 



I s 

_ o 
33 T3 
3 * — 
33 > 

1:5 O 

o c 

— 

c C. 

o a. 



«- -j o 
3 "3 2 

0 0 3 



JE to 

G. E 



-3 



o 

3 

O 



G. C 
X o 
O -3 
CJ o 

o ^ 
" S 

C6 C^ 
-n 



3 

CP 



ir p 3 
2 -5 "2 
5 .5 c 

c'S ^ 

— o .5 



“3 
— ' 3 



X, 



X 3 
- ^ = 



o — 

C 'C 



o >> 

Cl 0 = ^ — 

g .2P!5 1 
60 — ^ 5. 
E . — to 

3 O o p 



3 

O 

-3 



O 

to 



3T O 3 .2 3 

t: b o — - 

.= :: s o - 



3 o — 



v. •_ 
3 CO 



O 

60 



>, ■- 
3 O 
33 



O O 



.3 ? 



^g = 
5 « -2 
*- E* 

C6 O -X O 
w C, b 3 
x § 33 
O > CJ 



00 



_4 

O 

E 



00 



4 3 
04 O O 
° > C 
o o ^ 
•— 33 -» 

3 CJ 



■ — o 



o o 



3 


•5 


3 


33 

O 


O 


jj 


O 


v: 


v; 

60 


\r. 


.3 


3 


v: 


O 


X 




■ — 


— 


TZ 




3 


X 


O 

O 


£— 

£5 


vi 

vi 

c£ 


E 

c 


d 

1 — 

0 


y. 


3- 

"d 


CE 


3 


3 


c 

c 

CJ 


lo 1 

3 


CO 


0 

1— 


vi 

3 


d 


"3 


> 

0 


V? 


> 


O0 


3 


ir. 


• 


c_ 


U 


0 


3 


33 


• 


O 


1 





03 



oo 

^s» 






A 



ID 



CN 








o 

to 



Appendix B 




52 




Massachusetts Workplace Literacy Consortium Year 3 Evaluation, October, 1997 



NWLP, Wave 6, Year III Evaluation 

of the 

MASSACHUSETTS WORKPLACE LITERACY CONSORTIUM 



CPET Interview Probes 



I. Organization/Governance 

• With regard to the organization and functioning of the CPET, how would you 
describe the role of a) the DOE, and b) the program coordinators? 

• What were the major accomplishments of the CPET? 

II. Consortium 

• Do you have a sense of belonging to a consortium? 

• How much interaction or sharing of resources was there among the program 
coordinators? 

III. Institutionalizing Programs 

• At each of your sites, what have been the major forces for and against 
institutionalization of workplace literacy programs? 

• What role has the CPET or Consortium played in promoting institutionalization? 

IV. PET 

• How much variation is there among the PETs across your sites? 

• Based on your experiences, what are the characteristics of a good PET ? 
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Massachusetts Workplace Literacy Consortium Year 3 Evaluation, October, 1997 
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Massachusetts Workplace Literacy Consortium Year 3 Evaluation, October, 1997 



PET Site Visits 
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Appendix D 



O 

ERLC 



Planning and Evaluation Team Participation Chart. 



After the interview meeting, Coordinator should complete the chart below for each 
person not present who participated on the PET during the past 12 months. Please 

identify a title category for each participant using the following code: 

E = Education Provider W = Worker/Student 

M = Management T = Teacher 

S = Supervisor O = Other (please specify) 

U = Union 



NAME 


TITLE CODE 
(See List Above) 


Currently on 
PET 


# Years / 
Months 
On PET 


# Of PET 
Meetings Attended 
in past 1 2 Months 






Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 










Y 


N 






. 




Y 


N 










Y 


N 
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NWLP, Wave 6, Year III Evaluation 
of the 

MASSACHUSETTS WORKPLACE LITERACY CONSORTIUM 

P.E.T. Survey 



The Massachusetts Workplace Literacy Consortium is now in its third and final year of funding through the 
National Workplace Literacy Program. The legislation that funds your program requires an external 
evaluation each year. The puipose of the evaluation is to determine how the Consortium is meeting its goals. 
We hope that your participation in the evaluation will help you improve your program. 

One focus of the evaluation is the effectiveness of the Planning and Evaluation Team (PET) model for 
administering your workplace education programs. This survey contains two sets of questions related to the 
organization and operation of your PET. The first set of questions contains group questions. PET members 
answered these questions during our visit to your PET meeting. Please respond to the group questions if you 
were unable to attend that PET meeting. You may also respond to the group questions to add to the 
information provided at the PET meeting. The second set of questions contains individual questions. We 
ask all PET participants to complete the individual questions. 

We ask you to answer all questions as completely as you can. We estimate that it will take you no more than 
20 minutes to complete the survey. 

Only the external evaluators will view your completed survey. We will not identify individuals in the 
reporting of evaluation results. We will use current position and amount of PET experience only to 
determine groupings for analyses. If you have any questions concerning the survey or the evaluation, please 
contact Charles DePascale, one of the external evaluators at (508) 287-5170. 

Thank you for taking the time to provide information about your PET. Please return your completed survey 
within the next week in the postage paid envelope provided to 

Charles DePascale 
DATA, Inc. 

PO Box 395 
Concord, MA 01742 
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GROUP QUESTIONS 

1 . For each of the following PET Governance Tasks, indicate (/ ) who participated in each task. 



PET Governance Tasks 


PET including 
Coordinator 


Coordinator 

alone 


Other 

(Specify) 


a. Develop project goals 








b. Develop project plan 








c. Review plan regularly 








d. Develop project activities 








e. Input into creating curriculum 








f. Evaluate the PET 








g. Develop an institutionalization plan 








h. Overcome barriers to participation in PET 








i. Overcome barriers to participation in classes 








j. Provide incentives to program participation 








k. Evaluate progress of student/worker 








1. Evaluate impact of program on the worksite 








m. Evaluate stability of the partnership 








n. Other 









2. The PET model was designed to be a demoeratic participatory approach to governance of a workplace education program. 

This model may not be appropriate for all individuals in a PET or for all companies/unions or volunteer organizations involved 
in the MA Workplace Literacy Consortium. Please comment on the following 

A. Does a team approach fit in well with your organization? 



B. How has this democratic and open team approach to decision-making worked for your PET? (For example: Do 
workers and supervisors each have equal voices in decision making ? Has the model been myth or reality for your 
PET?) 
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3. What are the key factors that have contributed to the success of your workplace education program? 



4. To what extent is the success of your workplace education program attributable to your PET? (Please comment) 

1 2 3 4 5 

no connection 100% result of 

PET 



5. What have been the barriers to the workplace education program? 



6. To what extent has this PET successfully overcome the barriers ? Please comment. 

I 2 3 

no success 



;> 

100% successful 



7. What additional support and assistance could your PET have received from the following groups: 



GROUP 


Please indicated what additional support was needed 


Department of Education 




Consortium PET 




PET Coordinator/ 
Education Provider 




Other PET members 




Business 




Union 




Other, please specify 





8. If you were starting a new workplace education program, would you have a planning anil evaluation team ? 



0 




NWLP, Wave 6, Year III Evaluation: PET SURVEY 



yes 



no. 



Why? 



9. What would you keep or change about your “future program” PET? 



1 0. What steps, if any, is your PET taking to ensure the continuation of your workplace education program at the conclusion of the 
current funding? 



1 1 . What additional comments would you like to make regarding the organization, operatiof . or effectiveness of your PET? 



tKJC 
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Individual Questions 



Current Position 

Site Months on Planning and Evaluation Team. 



1. How many PET meetings have you attended during the past twelve months? 

2. Indicate your level of involvement in the following PET activities on the following 4-point scale 

1 = no involvement 2 = slight/limited involvement 

3 = moderate involvement 4 = total involvement 



PET Governance Tasks 


Level of Involvement 


1 . Develop project goals 


1 


2 


3 


4 


2. Develop project plan 


1 


2 


3 


4 


3. Review plan regularly 


1 


2 


3 


4 


4. Develop project activities 


1 


2 


3 


4 


5. Input into creating curriculum 


1 


2 


3 


4 


6. Evaluate the PET 


1 


2 


3 


4 


7. Develop an institutionalization plan 


1 


2 


3 


4 


8. Overcome barriers to participation in PET 


1 


2 


3 


4 


9. Overcome barriers to participation in classes 


1 


2 


3 


4 


10. Provide incentives to program participation 


1 


2 


3 


4 


1 1. Evaluate progress of student/worker 


I 


2 


3 


4 


1 2. Evaluate impact of program on the worksite 


1 


2 


3 


4 


1 3. Evaluate stability of the partnership 


1 


2 


3 


4 


14. Other 


1 


2 


3 


4 



3. Do you feel that all relevant stakeholders (i.e., educators, workers/students, unions, management, 
supervisors, volunteers, etc.) are able to fully participate in PET discussions and decision-making ? 

yes no 

If no, which stakeholders did not fully participate? 



4. Do students have a full voice in the activities of your PET? 
yes no 

If no, why? 
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5. What does your PET do to encourage student participation and input into PET discussions and 
decisions? 



6. How would you describe the role of the representative of the education provider (PET Project 
Coordinator) on your PET? 



7. Based on your PET experiences, what are the major pros and cons of the PET model in enhancing the 
success of a workplace education program? 

Pros Cons 



8. Would you want to be part of a future Planning and Evaluation Team? 
yes no 

Why or why not? 



9. Has your PET received sufficient support and assistance from the following groups: 
1= insufficient support in all areas 2=suficient support in some areas 

3=sufficient support in most areas 4=sufficient support in all areas 



GROUP 


QUALITY OF SUPPORT 


Department of Education 


12 3 4 


Consortium PET 


12 3 4 


PET Coordinator/Education Provider 


12 3 4 


Other PET members 


12 3 4 


Business 


12 3 4 


Union 


12 3 4 


Other 


12 3 4 




NWLP, Wave 6, Year III Evaluation 
of the 

MASSACHUSETTS WORKPLACE LITERACY CONSORTIUM 

PET Meeting Observation Form 



Note: 

• Table/Chair Location 

• Obstacles 

• Lighting 

• Doors/Windows 



I. Describe the meeting room in terms of 

A. Style (board room, conference room, classroom, etc.) 

B. Location 

C. Ambience (noise level, carpeting, comfort, etc.) 



II. Speakers 



Indicate (/ ) which people speak at meeting 


Program Coordinator 


Supervisory Staff 


Labor Union Representative 


HR Manager 


Student/Worker 


Other (specify) 


Corporate Management 


Teacher 



III. Materials 



Indicate (/ ) materials available at meeting 


Agenda 

^ 


Meeting Minutes 

Sc 


Teacher Reports 


Other Reports 



IV. General Notes on Meeting Including 

A. Chair of meeting 

B. Organization of meeting 

C. Interactions among members 
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